Tikhonov-Type Regularization for Restricted Boltzmann Machines
نویسندگان
چکیده
In this paper, we study a Tikhonov-type regularization for restricted Boltzmann machines (RBM). We present two alternative formulations of the Tikhonov-type regularization which encourage an RBM to learn a smoother probability distribution. Both formulations turn out to be combinations of the widely used weight-decay and sparsity regularization. We empirically evaluate the effect of the proposed regularization schemes and show that the use of them could help extracting better discriminative features with sparser hidden activation probabilities.
منابع مشابه
Sparse Group Restricted Boltzmann Machines
Since learning in Boltzmann machines is typically quite slow, there is a need to restrict connections within hidden layers. However, the resulting states of hidden units exhibit statistical dependencies. Based on this observation, we propose using l1/l2 regularization upon the activation probabilities of hidden units in restricted Boltzmann machines to capture the local dependencies among hidde...
متن کاملElastic regularization in restricted Boltzmann machines: Dealing with $p\gg N$
Restricted Boltzmann machines (RBMs) are endowed with the universal power of modeling (binary) joint distributions. Meanwhile, as a result of their confining network structure, training RBMs confronts less difficulties (compared with more complicated models, e.g., Boltzmann machines) when dealing with approximation and inference issues. However, in certain computational biology scenarios, such ...
متن کاملInductive Principles for Learning Restricted Boltzmann Machines (DRAFT: August 25, 2010)
We explore the training and usage of the Restricted Boltzmann Machine for unsupervised feature extraction. We investigate the many different aspects involved in their training, and by applying the concept of iterate averaging we show that it is possible to greatly improve on state of the art algorithms. We also derive estimators based on the principles of pseudo-likelihood, ratio matching, and ...
متن کاملA numerical approach for solving a nonlinear inverse diusion problem by Tikhonov regularization
In this paper, we propose an algorithm for numerical solving an inverse non-linear diusion problem. In additional, the least-squares method is adopted tond the solution. To regularize the resultant ill-conditioned linear system ofequations, we apply the Tikhonov regularization method to obtain the stablenumerical approximation to the solution. Some numerical experiments con-rm the utility of th...
متن کاملInvestigating Convergence of Restricted Boltzmann Machine Learning
Restricted Boltzmann Machines are increasingly popular tools for unsupervised learning. They are very general, can cope with missing data and are used to pretrain deep learning machines. RBMs learn a generative model of the data distribution. As exact gradient ascent on the data likelihood is infeasible, typically Markov Chain Monte Carlo approximations to the gradient such as Contrastive Diver...
متن کامل